Image processing apparatus, image processing method, and program

ABSTRACT

A decision section decides, using a current image captured by a camera and a past image captured by the camera, whether or not it is necessary to calibrate images captured by the camera. The present disclosure is applicable, for example, to a vehicle-mounted camera system that includes an automobile vehicle, two cameras, mounted to a front roof of the automobile vehicle such that imaging regions overlap, and an image processing apparatus, and so on.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, animage processing method, and a program, and more particularly, to animage processing apparatus, an image processing method, and a programthat permit decision as to whether or not camera calibration isnecessary.

BACKGROUND ART

In image processing carried out by linking images captured by aplurality of cameras, it is important to control, with high accuracy,relationships between three dimensional (3D) positions and orientations(attitudes) of the cameras. Also, the relationships between 3D positionsand orientations of the cameras change depending on impact, temperature,lapse of time, and so on. Therefore, control parameters of theserelationships need to be updated regularly.

Correction of images captured by cameras is known as an approach tocontrolling relationships between 3D positions and orientations ofcameras such that given relationships are present. As an approach todetermining parameters used for image correction, i.e., a cameracalibration method, for example, correction parameters are found suchthat desired images are produced as a result of capture of a specificpattern with the respective cameras. This method requires shooting of aspecific pattern, making it difficult to update the parametersregularly.

For this reason, there is proposed a method of determining parameterssuch that similarity of a known object in each image is enhanced byusing images captured by a plurality of cameras under normal use withoutusing a specific pattern (refer, for example, to PTL 1). This methodpermits determination of parameters under normal use, thereby allowingfor regular updating of the parameters. Also, there is conceived amethod of determining parameters by shooting an arbitrary scene.

CITATION LIST Patent Literature [PTL 1]

JP 2012-75060A

SUMMARY Technical Problem

Incidentally, camera calibration requires relatively much processing.Therefore, it is desired to reduce the amount of processing by decidingwhether or not camera calibration is necessary and calibrating thecamera only when needed.

The present disclosure has been devised in light of the above problem,and it is an object of the present disclosure to permit decision as towhether or not camera calibration is necessary.

Solution to Problem

An image processing apparatus of an aspect of the present disclosureincludes a decision section that decides, using a first image capturedby a first imaging section at a first time and a second image capturedby the first imaging section at a second time later than the first time,whether or not it is necessary to calibrate the first imaging section.

An image processing method and a program of an aspect of the presentdisclosure are associated with the image processing apparatus of anaspect of the present disclosure.

In an aspect of the present disclosure, whether or not it is necessaryto calibrate a first imaging section is decided using a first imagecaptured by the first imaging section at a first time and a second imagecaptured by the first imaging section at a second time later than thefirst time.

Advantageous Effects of Invention

According to an aspect of the present disclosure, it is possible toperform image processing. Also, according to an aspect of the presentdisclosure, it is possible to decide whether or not camera calibrationis necessary.

It should be noted that the effects described herein are not necessarilylimited and may be any one of the effects described in the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an overview of a first embodiment of avehicle-mounted camera system to which the present disclosure isapplied.

FIG. 2 is a block diagram illustrating a configuration example of animage processing apparatus mounted to a vehicle depicted in FIG. 1.

FIG. 3 is a block diagram illustrating a configuration example of anamount-of-travel estimation section depicted in FIG. 2.

FIG. 4 is a block diagram illustrating a configuration example of aposition detection section depicted in FIG. 2.

FIG. 5 is a flowchart describing image processing of the imageprocessing apparatus depicted in FIG. 2.

FIG. 6 is a flowchart describing details of a calibration decisionprocess depicted in FIG. 5.

FIG. 7 is a flowchart describing details of a camera relationshiprecognition process depicted in FIG. 5.

FIG. 8 is a diagram illustrating an overview of a second embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

FIG. 9 is a block diagram illustrating a configuration example of theposition detection section of the vehicle-mounted camera system depictedin FIG. 8.

FIG. 10 is a diagram illustrating an overview of a third embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

FIG. 11 is a diagram illustrating an overview of a fourth embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

FIG. 12 is a block diagram illustrating a configuration example of theimage processing apparatus mounted to a vehicle depicted in FIG. 11.

FIG. 13 is a flowchart describing the calibration decision process ofthe image processing apparatus depicted in FIG. 12.

FIG. 14 is a diagram illustrating an overview of a fifth embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

FIG. 15 is a block diagram illustrating a hardware configuration exampleof a computer.

FIG. 16 is a block diagram illustrating an example of a schematicconfiguration of a vehicle control system.

FIG. 17 is an explanatory diagram illustrating examples of installationpositions of an out-vehicle information detection section and an imagingsection.

DESCRIPTION OF EMBODIMENTS

A description will be given below of modes for carrying out the presentdisclosure (hereinafter referred to as embodiments). It should be notedthat a description will be given in the following order:

1. First Embodiment: Vehicle-Mounted Camera System (FIG. 1 to FIG. 7) 2.Second Embodiment: Vehicle-Mounted Camera System (FIGS. 8 and 9) 3.Third Embodiment: Vehicle-Mounted Camera System (FIG. 10) 4. FourthEmbodiment: Vehicle-Mounted Camera System (FIG. 11 to FIG. 13) 5. FifthEmbodiment: Vehicle-Mounted Camera System (FIG. 14) 6. Sixth Embodiment:Computer (FIG. 15) 7. Seventh Embodiment: Vehicle Control System (FIGS.16 and 17) First Embodiment (Overview of First Embodiment ofVehicle-Mounted Camera System)

FIG. 1 is a diagram illustrating an overview of a first embodiment of avehicle-mounted camera system to which the present disclosure isapplied.

A vehicle-mounted camera system 10 depicted in FIG. 1 includes a vehicle11, cameras 12 and 13, and so on. It should be noted that, in thepresent specification, front, back, right, and left as one faces thedirection of travel of the vehicle 11 during normal driving will bereferred to as front, back, right, and left of the vehicle 11.

The camera 12 (first imaging section) and the camera 13 (second imagingsection) are mounted to a roof front of the vehicle 11 such that imagingregions overlap. In the example depicted in FIG. 1, a road sign 21, aknown object ahead of the vehicle 11, is included in the imaging regionsof the cameras 12 and 13.

Although the road sign 21 is a known object in the first embodiment, aknown object may be an object on the road other than a road sign or anobject of the vehicle 11 such as number plate and emblem as long as itis an object of known size and shape.

The image processing apparatus, not depicted, which is mounted to thevehicle 11 estimates 3D positions and orientations of the road sign 21with respect to each of the cameras 12 and 13.

It should be noted that the 3D position with respect to the camera 12(camera 13) is, for example, a position in x, y, and z directions when agiven position (e.g., center) of an imaging plane of the camera 12(camera 13) is assumed to be an origin and when horizontal and verticaldirections of the imaging plane thereof and the direction vertical tothe imaging plane are assumed to be the x, y, and z directions,respectively. Also, the orientation with respect to the camera 12(camera 13) is a rotation angle about these x, y, and z directions.

The image processing apparatus recognizes relationships between 3Dpositions and orientations of the cameras 12 and 13 on the basis of the3D positions and orientations of the road sign 21 with respect to eachof the cameras 12 and 13, and performs calibration of the cameras 12 and13 on the basis of these relationships.

The relationship between 3D positions of the cameras 12 and 13 refers,for example, to a position of a given position of the imaging plane ofthe other of the cameras 12 and 13 in the x, y, and z directions when agiven position (e.g., center) of the imaging plane of one of the cameras12 and 13 is assumed to be the origin and when the horizontal andvertical directions of the imaging plane thereof and the directionvertical to the imaging plane are assumed to be the x, y, and zdirections, respectively. Also, the relationship between orientations ofthe cameras 12 and 13 refers, for example, to a rotation angle of theimaging plane of the other of the cameras 12 and 13 about these x, y,and z directions.

(Configuration Example of Image Processing Apparatus)

FIG. 2 is a block diagram illustrating a configuration example of theimage processing apparatus mounted to the vehicle 11 depicted in FIG. 1.

An image processing apparatus 40 depicted in FIG. 2 includes anamount-of-travel estimation section 41, a decision section 42, anamount-of-travel estimation section 43, a decision section 44, aposition detection section 45, and a correction section 46.

Images captured by the cameras 12 and 13 are input to the imageprocessing apparatus 40. Images captured by the camera 12 are suppliedto the amount-of-travel estimation section 41, the amount-of-travelestimation section 43, the position detection section 45, and thecorrection section 46, and images captured by the camera 13 are suppliedto the amount-of-travel estimation section 41, the amount-of-travelestimation section 43, the position detection section 45, and thecorrection section 46. Also, imaging parameters are input to the imageprocessing apparatus 40 from the cameras 12 and 13 and supplied to theposition detection section 45.

It should be noted that imaging parameters are, for example, internalparameters such as horizontal and vertical magnifying ratios on thebasis of focal distance, pixel size, and so on that are used when aposition on an image is converted into a position in the 3D space of thereal world. Details of internal parameters are described, for example,in PTL 1.

The amount-of-travel estimation section 41 of the image processingapparatus 40 estimates, using images at different times (frames)supplied from the camera 12, amounts of travel of the 3D position andorientation of the camera 12 between those times and supplies theamounts of travel to the decision section 42.

The decision section 42 decides whether or not it is necessary tocalibrate the camera 12 on the basis of the amounts of travel suppliedfrom the amount-of-travel estimation section 41 and a speed of thevehicle 11 measured by a speedometer, not depicted, which is mounted tothe vehicle 11. The decision section 42 supplies a decision result tothe position detection section 45 and the correction section 46.

The amount-of-travel estimation section 43 estimates, using images atdifferent times (frames) supplied from the camera 13, amounts of travelof the 3D position and orientation of the camera 13 during those timesand supplies the amounts of travel to the decision section 44.

The decision section 44 decides whether or not it is necessary tocalibrate the camera 13 on the basis of the amounts of travel suppliedfrom the amount-of-travel estimation section 43 and a speed of thevehicle 11 measured by a speedometer, not depicted, which is mounted tothe vehicle 11. The decision section 44 supplies a decision result tothe position detection section 45 and the correction section 46.

In a case where calibration is necessary according to the decisionresult supplied from at least one of the decision section 42 and thedecision section 44, the position detection section 45 detects the roadsign 21 as a common known object included in the images supplied fromthe cameras 12 and 13.

Then, the position detection section 45 estimates the 3D positions andorientations of the road sign 21 with respect to each of the cameras 12and 13, on the basis of the imaging parameters supplied from each of thecameras 12 and 13. The position detection section 45 recognizes therelationships between 3D positions and orientations of the cameras 12and 13 on the basis of these 3D position and orientation. The positiondetection section 45 supplies these relationships to the correctionsection 46.

The correction section 46 determines correction parameters used tocorrect images of at least one of the cameras 12 and 13 on the basis ofthe relationships between the cameras 12 and 13 supplied from theposition detection section 45 and the decision results supplied from thedecision section 42 and the decision section 44 and retains (updates)the correction parameters. The correction section 46 corrects the imagessupplied from the cameras 12 and 13 using the retained correctionparameters and outputs the corrected images. Also, the retainedcorrection parameter for the camera 12 is read by the amount-of-travelestimation section 41 and used to estimate the amount of travel of thecamera 12. The retained correction parameter for the camera 13 is readby the amount-of-travel estimation section 43 and used to estimate theamount of travel of the camera 13.

The correction parameters can be determined, for example, such that theorientations of the cameras 12 and 13 and the positions thereof in the yand z directions are the same. In this case, the images captured by thecameras 12 and 13 are parallel to each other as a result of correctionby the correction section 46.

(Configuration Example of Amount-of-Travel Estimation Section)

FIG. 3 is a block diagram illustrating a configuration example of theamount-of-travel estimation section 41 depicted in FIG. 2.

The amount-of-travel estimation section 41 includes an image correctionsection 61, an image correction section 62, a feature point detectionsection 63, a parallax detection section 64, a position calculationsection 65, a feature quantity calculation section 66, a map informationstorage section 67, a motion matching section 68, and anamount-of-travel calculation section 69.

The image correction section 61 corrects an image supplied from thecamera 12 on the basis of the correction parameter for the camera 12retained by the correction section 46 depicted in FIG. 2 such that theimage faces the same direction as the image supplied from the camera 13.The image correction section 61 supplies the corrected image to theparallax detection section 64 and the motion matching section 68 as aleft image.

The image correction section 62 corrects an image supplied from thecamera 13 on the basis of the correction parameter for the camera 13retained by the correction section 46 depicted in FIG. 2 such that theimage faces the same direction as the image supplied from the camera 12.The image correction section 62 supplies the corrected image to thefeature point detection section 63 as a right image.

The feature point detection section 63 detects feature points of theright image supplied from the image correction section 62. The featurepoint detection section 63 supplies, to the parallax detection section64 and the feature quantity calculation section 66, right feature pointposition information indicating the position of each feature pointdetected on the right image and the right image.

The parallax detection section 64 detects, from the left image suppliedfrom the image correction section 61, feature points, each correspondingto one of feature points of the right image, on the basis of rightfeature point position information and the right image supplied from thefeature point detection section 63. The parallax detection section 64supplies, to the position calculation section 65, left feature pointposition information indicating the position of each feature pointdetected on the left image and the right feature point positioninformation. Also, the parallax detection section 64 detects, for eachfeature point, a difference between the position indicated by the rightfeature point position information and the position indicated by theleft feature point position information as a stereo parallax andsupplies the stereo parallax of each feature point to the positioncalculation section 65.

The position calculation section 65 calculates the position of eachfeature point in the 3D space of the real world on the basis of thestereo parallaxes supplied from the parallax detection section 64, theright feature point position information, and the left feature pointposition information. The position calculation section 65 supplies, tothe feature quantity calculation section 66, 3D position informationindicating the position of each feature point in the 3D space of thereal world.

The feature quantity calculation section 66 calculates a featurequantity of each feature point on the basis of the right feature pointposition information and the right image supplied from the feature pointdetection section 63. The feature quantity calculation section 66 storesfeature point information including 3D position information and thefeature quantity of each feature point in the map information storagesection 67.

The motion matching section 68 reads, from the map information storagesection 67, feature point information of each feature point detectedfrom the past left and right images. The motion matching section 68detects, on the basis of the feature quantity of each feature pointincluded in the read feature point information and from the current leftimage supplied from the image correction section 61, a feature pointthat corresponds to that feature point. The motion matching section 68supplies, to the amount-of-travel calculation section 69, 3D positioninformation of each feature point included in the read feature pointinformation and left feature point position information for that featurepoint.

The amount-of-travel calculation section 69 estimates, on the basis of3D position information of each past feature point supplied from themotion matching section 68 and left feature point position informationof the current feature point that corresponds to that feature point,frame-to-frame amounts of travel of the 3D position and orientation ofthe camera 12. The amount-of-travel calculation section 69 supplies theestimated amounts of travel to the decision section 42 depicted in FIG.2.

It should be noted that although the amount-of-travel estimation section41 depicted in FIG. 3 calculated the 3D position of each feature pointon the basis of the stereo parallax, the 3D position of each featurepoint may be calculated on the basis of a movement parallax. In thiscase, the parallax detection section 64 detects a movement parallax ofeach feature point of the left image (right image) using left images(right images) at different times. Then, the position calculationsection 65 calculates the 3D position of each feature point on the basisof the movement parallax detected by the parallax detection section 64and the left feature point position information (right feature pointposition information).

Also, although not depicted, the amount-of-travel estimation section 43has the similar configuration as the amount-of-travel estimation section41 depicted in FIG. 3 except that the right image is input to the motionmatching section 68 rather than the left image. It should be noted that,here, both the amount-of-travel estimation section 41 and the decisionsection 42 and both the amount-of-travel estimation section 43 and thedecision section 44 are provided separately so that the need forcalibration of the cameras 12 and 13 is decided separately. However, ina case where the amount-of-travel estimation section 41 is configured asdepicted in FIG. 3, the decision results as to the need for calibrationof the cameras 12 and 13 are the same. Therefore, only theamount-of-travel estimation section 41 and the decision section 42 maybe provided. In this case, the decision section 42 decides whether ornot it is necessary to calibrate both the cameras 12 and 13 on the basisof the amounts of travel supplied from the amount-of-travel estimationsection 41 and the speed of the vehicle 11 measured by the speedometer,not depicted, which is mounted to the vehicle 11. Also, theamount-of-travel estimation section 41 may estimate amounts of travelusing movement parallax rather than stereo parallax. In this case, boththe amount-of-travel estimation section 41 and the decision section 42and both the amount-of-travel estimation section 43 and the decisionsection 44 are provided separately, and whether or not it is necessaryto calibrate the cameras 12 and 13 is decided separately.

(Configuration Example of Position Detection Section)

FIG. 4 is a block diagram illustrating a configuration example of theposition detection section 45 depicted in FIG. 2.

The position detection section 45 depicted in FIG. 4 includes adictionary section 80, a feature point detection section 81, a matchingsection 82, an estimation section 83, a feature point detection section84, a matching section 85, an estimation section 86, and a recognitionsection 87.

The dictionary section 80 retains feature quantities of a plurality offeature points of the road sign 21, a known object.

The feature point detection section 81 detects feature points from theimage supplied from the camera 12 for use as feature point candidates ofa known object in response to decision results supplied from thedecision sections 42 and 44 depicted in FIG. 2. The feature pointdetection section 81 supplies feature quantities of the feature pointcandidates of the known object and two dimensional (2D) positions on theimage to the matching section 82.

The matching section 82 reads the feature quantities of the plurality offeature points of the road sign 21 from the dictionary section 80. Thematching section 82 performs matching, for each read feature point ofthe road sign 21, between the feature quantity of that feature point andthe feature quantity of the feature point candidate supplied from thefeature point detection section 81, selecting the feature pointcandidate with the highest similarity, as a feature point of the roadsign 21. The matching section 82 supplies, to the estimation section 83,the 2D position of the selected feature point of the road sign 21 on theimage.

It should be noted, however, that if, for example, the highestsimilarity of a given number of feature points or more of the featurepoints of the road sign 21 is equal to a threshold or less, the matchingsection 82 decides that the road sign 21 does not exist in the imagesupplied from the camera 12 and does not supply anything to theestimation section 83.

The estimation section 83 finds a 3D position of each feature point ofthe road sign 21 with respect to the camera 12 on the basis of the 2Dposition of each feature point of the road sign 21 on the image suppliedfrom the matching section 82 and the imaging parameters supplied fromthe camera 12. The estimation section 83 estimates the 3D position andorientation of the road sign 21 with respect to the camera 12 on thebasis of the 3D position of each of the feature points of the road sign21 with respect to the camera 12 and supplies the 3D position andorientation to the recognition section 87.

The feature point detection section 84 detects feature points from theimage supplied from the camera 13 for use as feature point candidates ofa known object in response to decision results supplied from thedecision sections 42 and 44 depicted in FIG. 2. The feature pointdetection section 84 supplies feature quantities of the feature pointcandidates of the known object and 2D positions on the image to thematching section 85.

The matching section 85 reads the feature quantities of the plurality offeature points of the road sign 21 from the dictionary section 80. Thematching section 85 performs matching, for each read feature point ofthe road sign 21, the feature quantity of that feature point and thefeature quantity of the feature point candidate supplied from thefeature point detection section 84, selecting the feature pointcandidate with the highest similarity, as a feature point of the roadsign 21. The matching section 85 supplies, to the estimation section 86,the 2D position of the selected feature point of the road sign 21 on theimage.

It should be noted, however, that if, for example, the highestsimilarity of a given number of feature points or more of the featurepoints of the road sign 21 is equal to a threshold or less, the matchingsection 85 decides that the road sign 21 does not exist in the imagesupplied from the camera 13 and does not supply anything to theestimation section 86.

The estimation section 86 finds a 3D position of each feature point ofthe road sign 21 with respect to the camera 13 on the basis of the 2Dposition of each feature point of the road sign 21 on the image suppliedfrom the matching section 82 and the imaging parameters supplied fromthe camera 13. The estimation section 86 estimates the 3D position andorientation of the road sign 21 with respect to the camera 13 on thebasis of the 3D position of each of the feature points of the road sign21 with respect to the camera 13 and supplies the 3D position andorientation to the recognition section 87.

The recognition section 87 recognizes relationships between the 3Dpositions and orientations of the cameras 12 and 13 on the basis of the3D position and orientation of the road sign 21 with respect to thecamera 12 from the estimation section 83 and the 3D position andorientation of the road sign 21 with respect to the camera 13 from theestimation section 86. The recognition section 87 supplies therecognized relationships between the 3D positions and orientations ofthe cameras 12 and 13 to the correction section 46 depicted in FIG. 2.

(Description of Processes Handled by Image Processing Apparatus)

FIG. 5 is a flowchart describing image processing of the imageprocessing apparatus 40 depicted in FIG. 2. This image processingstarts, for example, when images captured by the cameras 12 and 13 areinput to the image processing apparatus 40.

In step S11 depicted in FIG. 5, the image processing apparatus 40performs a calibration decision process that decides whether or not itis necessary to calibrate the cameras 12 and 13. Details of thiscalibration decision process will be described with reference to FIG. 6which will be described later.

In step S12, the position detection section 45 decides whether it isnecessary to calibrate at least one of the cameras 12 and 13 on thebasis of decision results supplied from the decision section 42 and thedecision section 44.

Specifically, in a case where calibration is necessary according to thedecision result supplied from at least one of the decision section 42and the decision section 44, the position detection section 45 decidesthat it is necessary to calibrate at least one of the cameras 12 and 13.On the other hand, in a case where calibration is not necessaryaccording to both of the decision results supplied from the decisionsection 42 and the decision section 44, the position detection section45 decides that it is not necessary to calibrate both the cameras 12 and13.

In a case where it is decided in step S12 that at least one of thecameras 12 and 13 needs calibration, the position detection section 45performs, in step S13, a camera relationship recognition process thatrecognizes relationships between 3D positions and orientations of thecameras 12 and 13. Details of this camera relationship recognitionprocess will be described with reference to FIG. 7 which will bedescribed later.

In step S14, the correction section 46 decides whether it is necessaryto calibrate the camera 12 on the basis of the decision result suppliedfrom the decision section 42.

In a case where it is decided in step S14 that the camera 12 needscalibration, the process proceeds to step S15. In step S15, thecorrection section 46 determines a correction parameter used to correctimages of the camera 12 on the basis of the relationships between the 3Dpositions and orientations of the cameras 12 and 13 supplied from theposition detection section 45 and retains (updates) the correctionparameter. Then, the process proceeds to step S16.

On the other hand, in a case where it is decided in step S14 that thecamera 12 does not need calibration, the process proceeds to step S16.

In step S16, the correction section 46 decides whether it is necessaryto calibrate the camera 13 on the basis of the decision result suppliedfrom the decision section 44.

In a case where it is decided in step S16 that the camera 13 needscalibration, the process proceeds to step S17. In step S17, thecorrection section 46 determines a correction parameter used to correctimages of the camera 13 on the basis of the relationships between the 3Dpositions and orientations of the camera 13 and the camera 13 suppliedfrom the position detection section 45 and retains (updates) thecorrection parameter. Then, the process proceeds to step S18.

On the other hand, in a case where it is decided in step S16 that thecamera 13 does not need calibration, the process proceeds to step S18.

Also, in a case where it is decided in step S12 that both the cameras 12and 13 do not need calibration, the process proceeds to step S18.

In step S18, the correction section 46 corrects the images supplied fromthe cameras 12 and 13 using the retained correction parameters, outputsthe corrected images, and terminates the process.

FIG. 6 is a flowchart describing, of the calibration decision process instep S11 in FIG. 5, details of the calibration decision process for thecamera 12 to decide whether or not it is necessary to calibrate thecamera 12.

In step S31 depicted in FIG. 6, the image correction section 61 correctsthe image supplied from the camera 12 on the basis of the correctionparameter of the camera 12 retained by the correction section 46depicted in FIG. 2 such that the image faces the same direction as theimage supplied from the camera 13. The image correction section 61supplies the corrected image to the parallax detection section 64 andthe motion matching section 68 as a left image. Also, the imagecorrection section 62 corrects the image supplied from the camera 13 onthe basis of the correction parameter of the camera 13 retained by thecorrection section 46 depicted in FIG. 2 such that the image faces thesame direction as the image supplied from the camera 12. The imagecorrection section 62 supplies the corrected image to the feature pointdetection section 63 as a right image.

In step S32, the feature point detection section 63 detects featurepoints of the right image supplied from the image correction section 62.The feature point detection section 63 supplies right feature pointposition information of each of the detected feature points and theright image to the parallax detection section 64 and the featurequantity calculation section 66.

In step S33, the parallax detection section 64 detects a feature pointcorresponding to each feature point of the right image from the leftimage supplied from the image correction section 61 on the basis of theright feature point position information and the right image suppliedfrom the feature point detection section 63. The parallax detectionsection 64 supplies, to the position calculation section 65, leftfeature point position information of each of the detected featurepoints and the right feature point position information.

In step S34, the parallax detection section 64 detects a differencebetween the position indicated by the right feature point positioninformation and the position indicated by the left feature pointposition information as a stereo parallax and supplies the stereoparallax of each of the feature points to the position calculationsection 65.

In step S35, the position calculation section 65 calculates a positionof each of the feature points in the 3D space of the real world on thebasis of the stereo parallax supplied from the parallax detectionsection 64, the right feature point position information and the leftfeature point position information. The position calculation section 65supplies 3D position information of each feature point to the featurequantity calculation section 66.

In step S36, the feature quantity calculation section 66 calculates afeature quantity of each feature point on the basis of the right featurepoint position information and the right image supplied from the featurepoint detection section 63.

In step S37, the feature quantity calculation section 66 suppliesfeature point information including 3D position information of eachfeature point to the map information storage section 67 for storage.

In step S38, the motion matching section 68 reads, from the mapinformation storage section 67, feature point information of eachfeature point detected from past left and right images.

In step S39, the motion matching section 68 detects, on the basis of thefeature quantity of each feature point included in the read featurepoint information, a feature point corresponding to that feature pointfrom the current left image supplied from the image correction section61. The motion matching section 68 supplies, to the amount-of-travelcalculation section 69, the 3D position information of each featurepoint included in the read feature point information and left featurepoint position information corresponding to that feature point.

In step S40, the amount-of-travel calculation section 69 estimatesframe-to-frame amounts of travel of the 3D position and orientation ofthe camera 12 on the basis of the 3D position information and the leftfeature point position information supplied from the motion matchingsection 68. The amount-of-travel calculation section 69 supplies theestimated amounts of travel to the decision section 42.

In step S41, the decision section 42 decides whether or not it isnecessary to calibrate the camera 12 on the basis of the amounts oftravel of the camera 12 supplied from the amount-of-travel estimationsection 41 and the speed of the vehicle 11 measured by the speedometer,not depicted, which is mounted to the vehicle 11. The decision section42 supplies the decision result to the position detection section 45 andthe correction section 46.

It should be noted that, of the calibration decision process, thecalibration decision process for the camera 13 that decides whether ornot it is necessary to calibrate the camera 13 is not depicted becausethis process is similar to the calibration decision process for thecamera 12 depicted in FIG. 6 except for the processes from step S39 tostep S41.

In the calibration decision process for the camera 13, feature pointsare detected from the right image in step S39, amounts of travel of the3D position and orientation of the camera 13 are estimated in step S40,and whether or not the camera 13 needs calibration is decided in stepS41.

FIG. 7 is a flowchart describing details of the camera relationshiprecognition process in step S13 in FIG. 5.

In step S51 depicted in FIG. 7, the feature point detection section 81(FIG. 4) detects feature points from the image supplied from the camera12 for use as feature point candidates of the road sign 21. The featurepoint detection section 81 supplies feature quantities of the featurepoint candidates of the road sign 21 and 2D positions on the image tothe matching section 82.

In step S52, the matching section 82 performs matching, for each featurepoint of the road sign 21 whose feature quantity is retained by thedictionary section 80, between the feature quantity of that featurepoint and the feature quantity of the feature point candidate of theroad sign 21 supplied from the feature point detection section 81. Thematching section 82 selects the feature point candidate of the road sign21 with the highest similarity obtained as a result of the matching as afeature point of the road sign 21.

In step S53, the matching section 82 decides whether the road sign isincluded in the image supplied from the camera 12. For example, in acase where, of the selected feature points of the road sign 21, thenumber of feature points whose similarities are equal to a threshold orless is smaller than a given number, the matching section 82 decidesthat the road sign is included in the image supplied from the camera 12,and if the number of feature points is equal to the given number orlarger, the matching section 82 decides that the road sign is notincluded.

In a case where it is decided in step S53 that the road sign is includedin the image supplied from the camera 12, the matching section 82supplies 2D positions of the selected feature points of the road sign 21on the image to the estimation section 83.

Then, in step S54, the estimation section 83 estimates the 3D positionand orientation of the road sign 21 with respect to the camera 12 on thebasis of the 2D positions of the feature points of the road sign 21 onthe image supplied from the matching section 82 and the imagingparameters supplied from the camera 12. The estimation section 83supplies, to the recognition section 87, the estimated 3D position andorientation of the road sign 21 with respect to the camera 12.

In step S55, the feature point detection section 84 detects featurepoints from the image supplied from the camera 13 for use as featurepoint candidates of the road sign 21. The feature point detectionsection 84 supplies feature quantities of the feature point candidatesof the road sign 21 and 2D positions on the image to the matchingsection 85.

In step S56, the matching section 85 performs matching, for each featurepoint of the road sign 21 whose feature quantity is retained by thedictionary section 80, between the feature quantity of that featurepoint and the feature quantity of the feature point candidate of theroad sign 21 supplied from the feature point detection section 84. Thematching section 85 selects the feature point candidate of the road sign21 with the highest similarity obtained as a result of matching as afeature point of the road sign 21.

In step S57, the matching section 85 decides whether the road sign isincluded in the image supplied from the camera 13 as does the matchingsection 82. In a case where it is decided, in step S57, that the roadsign is included in the image supplied from the camera 13, the matchingsection 85 supplies 2D positions of the selected feature points of theroad sign 21 on the image to the estimation section 86.

Then, in step S58, the estimation section 86 estimates the 3D positionand orientation of the road sign 21 with respect to the camera 13 on thebasis of the 2D positions of the feature points of the road sign 21 onthe image supplied from the matching section 85 and the imagingparameters supplied from the camera 13. The estimation section 83supplies, to the recognition section 87, the estimated 3D position andorientation of the road sign 21 with respect to the camera 13.

In step S59, the recognition section 87 recognizes the relationshipsbetween the 3D positions and orientations of the cameras 12 and 13 onthe basis of the 3D position and orientation of the road sign 21 withrespect to the camera 12 and the 3D position and orientation of the roadsign 21 with respect to the camera 13. The recognition section 87supplies the relationships between the 3D positions and orientations ofthe cameras 12 and 13 to the correction section 46 depicted in FIG. 2.Then, the process returns to step S13 depicted in FIG. 5 and proceeds tostep S14.

On the other hand, in a case where it is decided, in step S53, that theroad sign is not included in the image supplied from the camera 12, orin a case where it is decided, in step S57, that the road sign is notincluded in the image supplied from the camera 13, the process proceedsto step S18 depicted in FIG. 5.

Thus, the image processing apparatus 40 estimates the 3D positions ofthe road sign 21, a known object often captured by the cameras 12 and 13under normal use of the vehicle-mounted camera system 10 and existing inmoderate numbers on roads, with respect to the cameras 12 and 13. Then,the image processing apparatus 40 recognizes the positional relationshipbetween the cameras 12 and 13 on the basis of that 3D position andcalibrates the cameras 12 and 13 on the basis of that relationship.

Therefore, it is possible to increase the calibration frequency of thecameras 12 and 13 as compared to calibration such as shooting a specificpattern that requires a special environment and equipment. Also, it isnot readily affected by the shooting environment, making it possible tocalibrate the cameras 12 and 13 with high accuracy. Further, calibrationof the cameras 12 and 13 requires only a limited amount of processing.

Also, the image processing apparatus 40 estimates an amount of travel ofthe camera 12 (camera 13) using images captured by the camera 12 (camera13) at different times and decides whether or not it is necessary tocalibrate the camera 12 (camera 13) on the basis of the amount oftravel.

Therefore, it is possible to calibrate the camera 12 (camera 13) only ifthe 3D position or orientation of the camera 12 (camera 13) deviates. Asa result, it is possible to reduce power consumption of the imageprocessing apparatus 40 and the capability and number of the processorsthat realize the image processing apparatus 40.

Second Embodiment (Overview of Second Embodiment of Vehicle-MountedCamera System)

FIG. 8 is a diagram illustrating an overview of a second embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

Of the components depicted in FIG. 8, the components identical to thosedepicted in FIG. 1 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate.

The configuration of a vehicle-mounted camera system 100 depicted inFIG. 8 differs from that of the vehicle-mounted camera system 10depicted in FIG. 1 in that there are two known objects, i.e., the roadsign 21 and a number plate 111.

The image processing apparatus, not depicted, mounted to the vehicle 11estimates 3D positions and orientations of the road sign 21 and thenumber plate 111 with respect to each of the cameras 12 and 13. Then,the image processing apparatus recognizes relationships between 3Dpositions and orientations of the cameras 12 and 13 on the basis of the3D positions and orientations of the road sign 21 and the number plate111 with respect to each of the cameras 12 and 13, and estimationaccuracy of the 3D positions and orientations. The image processingapparatus calibrates the cameras 12 and 13 on the basis of therelationships.

The configuration of the image processing apparatus of the secondembodiment is similar to that of the image processing apparatus 40depicted in FIG. 2 except for the configuration of the positiondetection section 45. Therefore, only the position detection section 45will be described below.

(Configuration Example of Position Detection Section)

FIG. 9 is a block diagram illustrating a configuration example of theposition detection section 45 of the vehicle-mounted camera system 100.

Of the components depicted in FIG. 9, the components identical to thosedepicted in FIG. 4 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate.

The configuration of the position detection section 45 depicted in FIG.9 differs from that of the configuration depicted in FIG. 4 in that adictionary section 120, a matching section 122, an estimation section123, a matching section 125, an estimation section 126, and arecognition section 127 are provided in place of the dictionary section80, the matching section 82, the estimation section 83, the matchingsection 85, the estimation section 86, and the recognition section 87.

The dictionary section 120 retains feature quantities of a plurality offeature points of the road sign 21 and the number plate 111, knownobjects.

The matching section 122 reads the feature quantities of the pluralityof feature points of the road sign 21 and the number plate 111 from thedictionary section 120. The matching section 122 selects, for each readfeature point of the road sign 21, a feature point of the road sign 21from among the feature point candidates supplied from the feature pointdetection section 81 as does the matching section 82 depicted in FIG. 4.

Also, the matching section 122 performs matching, for each read featurepoint of the number plate 111, between the feature quantity of thatfeature point and the feature quantity of the feature point candidatesupplied from the feature point detection section 81. The matchingsection 122 selects the feature point candidate with the highestsimilarity obtained as a result of the matching as a feature point ofthe number plate 111.

The matching section 122 supplies the 2D positions of the selectedfeature points of the road sign 21 and the number plate 111 on the imageto the estimation section 123. Also, the matching section 122 supplies,to the recognition section 127, similarity associated with the selectedfeature points of the road sign 21 and the number plate 111 asestimation accuracy by the estimation section 123.

It should be noted, however, that if, for example, the highestsimilarity of a given number of feature points or more of the featurepoints of the road sign 21 is equal to a threshold or less, the matchingsection 122 decides that the road sign 21 does not exist in the imagesupplied from the camera 12 and does not supply anything to theestimation section 123. It is also true for the number plate 111.

The estimation section 123 estimates the 3D position and orientation ofthe road sign 21 with respect to the camera 12 on the basis of the 2Dpositions of the feature points of the road sign 21 on the imagesupplied from the matching section 122 and the imaging parameterssupplied from the camera 12 as does the estimation section 83 depictedin FIG. 4. The estimation section 123 supplies, to the recognitionsection 127, the 3D position and orientation of the road sign 21 withrespect to the camera 12.

Also, the estimation section 123 finds a 3D position of each featurepoint of the number plate 111 with respect to the camera 12 on the basisof the 2D position of the feature point of the number plate 111 on theimage supplied from the matching section 122 and the imaging parametersof the camera 12. Then, the estimation section 123 estimates the 3Dposition and orientation of the number plate 111 with respect to thecamera 13 on the basis of the 3D position of each of the feature pointsof the number plate 111 with respect to the camera 12 and supplies the3D position and orientation to the recognition section 127.

The matching section 125 reads the feature quantities of the pluralityof feature points of the road sign 21 and the number plate 111 from thedictionary section 120. The matching section 125 selects, for each readfeature point of the road sign 21, a feature point of the road sign 21from among the feature point candidates supplied from the feature pointdetection section 84 as does the matching section 85 depicted in FIG. 4.

Also, the matching section 125 performs matching, for each read featurepoint of the number plate 111, between the feature quantity of thatfeature point and the feature quantity of the feature point candidatesupplied from the feature point detection section 84. The matchingsection 125 selects the feature point candidate with the highestsimilarity obtained as a result of the matching as a feature point ofthe number plate 111.

The matching section 125 supplies the 2D positions of the selectedfeature points of the road sign 21 and the number plate 111 on the imageto the estimation section 126. Also, the matching section 125 supplies,to the recognition section 127, similarity associated with the selectedfeature points of the road sign 21 and the number plate 111 as anestimation accuracy by the estimation section 126.

It should be noted, however, that if, for example, the highestsimilarity of a given number of feature points or more of the featurepoints of the road sign 21 is equal to a threshold or less, the matchingsection 125 decides that the road sign 21 does not exist in the imagesupplied from the camera 13 and does not supply anything to theestimation section 126. It is also true for the number plate 111.

The estimation section 126 estimates the 3D position and orientation ofthe road sign 21 with respect to the camera 13 on the basis of the 2Dpositions of the feature points of the road sign 21 on the imagesupplied from the matching section 125 and the imaging parameterssupplied from the camera 13 as does the estimation section 86 depictedin FIG. 4. The estimation section 126 supplies, to the recognitionsection 127, the 3D position and orientation of the road sign 21 withrespect to the camera 13.

Also, the estimation section 126 estimates the 3D position andorientation of the number plate 111 with respect to the camera 13 on thebasis of the 2D positions of the feature points of the number plate 111on the image supplied from the matching section 125 and the imagingparameters of the camera 13 and supplies the 3D position and orientationto the recognition section 127.

The recognition section 127 recognizes relationships between 3Dpositions and orientations of the cameras 12 and 13 on the basis of the3D positions and orientations of the road sign 21 and the number plate111 with respect to each of the cameras 12 and 13, and estimationaccuracies by the estimation sections 123 and 126.

Specifically, for example, the recognition section 127 selects the roadsign 21 or the number plate 111 associated with whichever is higher, theaverage estimation accuracy with respect to the road sign 21 or theaverage estimation accuracy with respect to the number plate 111 by theestimation sections 123 and 126. The recognition section 127 recognizesrelationships between 3D positions and orientations of the cameras 12and 13 on the basis of the 3D positions and orientations of the roadsign 21 or the number plate 111 with respect to each of the cameras 12and 13.

Alternatively, the recognition section 127 specifies a weight of theroad sign 21 on the basis of the average estimation accuracy withrespect to the road sign 21 by the estimation section 123 and theestimation section 126 such that the higher the average estimationaccuracy, the larger the weight. Also, the recognition section 127specifies a weight of the number plate 111 on the basis of the averageestimation accuracy with respect to the number plate 111 by theestimation section 123 and the estimation section 126 such that thehigher the average estimation accuracy, the larger the weight.

Then, the recognition section 127 recognizes, for each of the road sign21 and the number plate 111, relationships between 3D positions andorientations of the cameras 12 and 13 on the basis of the 3D positionsand orientations with respect to each of the cameras 12 and 13. Therecognition section 127 performs weighted addition of the 3D positionsand orientations of the cameras 12 and 13 recognized from the 3Dpositions and orientations of the road sign 21 and the number plate 111using the specified weights. The recognition section 127 recognizes theweighted addition results as the final relationships between 3Dpositions and orientations of the cameras 12 and 13.

The recognition section 127 supplies, to the correction section 46depicted in FIG. 2, the recognized relationships between 3D positionsand orientations of the cameras 12 and 13.

It should be noted that although, here, the estimation accuracies by theestimation sections 123 and 126 are, respectively, similarities in thematching sections 122 and 125, the estimation accuracies by theestimation sections 123 and 126 may be determined on the basis of 3Dpositions and orientations estimated by the estimation sections 123 and126.

Also, the camera relationship recognition process handled by theposition detection section 45 depicted in FIG. 9 is similar to thecamera relationship recognition process depicted in FIG. 7 except thatthe road sign 21 is replaced by the road sign 21 and the number plate111 and that estimation accuracies by the estimation sections 123 and126 are used in the process in step S59. Therefore, the description isomitted.

It should be noted that in a case where only one of the road sign 21 andthe number plate 111 is included in the images supplied from both thecameras 12 and 13, the recognition section 127 may recognizerelationships between 3D positions and orientations of the cameras 12and 13 on the basis of the 3D positions and orientations of that one ofthe road sign 21 and the number plate 111 with respect to each of thecameras 12 and 13, as does the recognition section 87 depicted in FIG.4.

Third Embodiment (Overview of Third Embodiment of Vehicle-Mounted CameraSystem)

FIG. 10 is a diagram illustrating an overview of a third embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

Of the components depicted in FIG. 10, the components identical to thosedepicted in FIG. 8 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate.

The configuration of a vehicle-mounted camera system 140 depicted inFIG. 10 differs from that of the vehicle-mounted camera system 100depicted in FIG. 8 in that not only the cameras 12 and 13 but also acamera 141 are mounted to the vehicle 11.

The camera 141 is mounted to a rear portion of the vehicle 11 to imageforward the vehicle 11. Each of the cameras 12, 13, and 141 has animaging region that overlaps with the imaging region of at least oneother camera.

In the example depicted in FIG. 10, the road sign 21, a known objectahead of the vehicle 11, is included in the imaging regions of thecameras 12 and 13. Also, the number plate 111, a known object ahead ofthe vehicle 11, is included in the imaging regions of the cameras 12, 13and 141.

In this case, the image processing apparatus, not depicted, which ismounted to the vehicle 11 estimates 3D positions and orientations of theroad sign 21 with respect to each of the cameras 12 and 13. Also, theimage processing apparatus estimates 3D positions and orientations ofthe number plate 111 with respect to each of the cameras 12, 13, and141.

The image processing apparatus recognizes relationships between 3Dpositions and orientations of the cameras 12, 13, and 141 on the basisof the 3D positions and orientations of the road sign 21 with respect toeach of the cameras 12 and 13, the 3D positions and orientations of thenumber plate 111 with respect to each of the cameras 12, 13, and 141,and estimation accuracies of the 3D positions and orientations.

Specifically, for example, the image processing apparatus recognizesrelationships between 3D positions and orientations of the cameras 12and 13 on the basis of the 3D positions and orientations of the roadsign 21 and the number plate 111 with respect to each of the cameras 12and 13, and estimation accuracies of the 3D positions and orientationsas does the image processing apparatus of the second embodiment.

Also, the image processing apparatus recognizes relationships between 3Dpositions and orientations of the camera 12 or 13 and the camera 141 onthe basis of the 3D position and orientation of the number plate 111with respect to either of the cameras 12 and 13, whichever has a higherestimation accuracy, and the 3D position and orientation of the numberplate 111 with respect to the camera 141.

Then, the image processing apparatus calibrates the cameras 12, 13, and141 on the basis of the relationships between 3D positions andorientations of the cameras 12 and 13 and the relationships between 3Dpositions and orientations of the camera 12 or 13 and the camera 141.

The image processing apparatus of the third embodiment is similar to theimage processing apparatus of the second embodiment except that not onlyprocesses for the cameras 12 and 13 but also processes for the camera141 are performed in the similar manner.

It should be noted that three amounts of travel are found by the imageprocessing apparatus of the third embodiment. Therefore, whether or notit is necessary to calibrate the cameras 12, 13, and 141 may be decidedon the basis of a distribution of three amounts of travel without usingthe speed of the vehicle 11 measured by the speedometer. In this case,it is decided that the camera 12, 13, or 141 whose amount of travel isan outlier of the distribution of three amounts of travel needscalibration.

Fourth Embodiment (Overview of Fourth Embodiment of Vehicle-MountedCamera System)

FIG. 11 is a diagram illustrating an overview of a fourth embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

Of the components depicted in FIG. 11, the components identical to thosedepicted in FIG. 1 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate.

The configuration of a vehicle-mounted camera system 160 depicted inFIG. 11 differs from that of the vehicle-mounted camera system 10depicted in FIG. 1 in that cameras 161 and 162 are provided in place ofthe cameras 12 and 13. It should be noted that FIG. 11 is a view of thevehicle 11 as seen from above.

The camera 161 (first imaging section) and the camera 162 (secondimaging section) are mounted such that the directions of their opticalaxes are approximately vertical to the direction of travel of thevehicle 11. In the example depicted in FIG. 11, the cameras 161 and 162are mounted to the right side roof of the vehicle 11.

In the vehicle-mounted camera system 160, the image processingapparatus, not depicted, which is mounted to the vehicle 11 decideswhether it is necessary to calibrate the cameras 161 and 162 on thebasis of the parallax of images captured by the cameras 161 and 162rather than amounts of travel.

(Configuration Example of Image Processing Apparatus)

FIG. 12 is a block diagram illustrating a configuration example of theimage processing apparatus mounted to the vehicle 11 depicted in FIG.11.

Of the components depicted in FIG. 12, the components identical to thosedepicted in FIG. 2 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate.

The configuration of an image processing apparatus 180 depicted in FIG.12 differs from that of the image processing apparatus 40 depicted inFIG. 2 in that an estimation section 181 is provided in place of theamount-of-travel estimation sections 41 and 43 and a decision section182 is provided in place of the decision sections 42 and 44.

The estimation section 181 includes a movement parallax estimationsection 191, a movement parallax estimation section 192, and a stereoparallax estimation section 193.

The movement parallax estimation section 191 of the estimation section181 retains an image captured by the camera 161 and input from thecamera 161. The movement parallax estimation section 191 (first movementparallax estimation section) estimates a movement parallax (motionparallax) of the current image using the retained past (first time)image and the current (second time) image. The movement parallaxestimation section 191 supplies the estimated movement parallax to thedecision section 182.

The movement parallax estimation section 192 retains an image capturedby the camera 162 and input from the camera 162. The movement parallaxestimation section 192 (second movement parallax estimation section)estimates a movement parallax (motion parallax) of the current imageusing the retained past (first time) image and the current (second time)image. The movement parallax estimation section 192 supplies theestimated movement parallax to the decision section 182.

The stereo parallax estimation section 193 estimates, using the currentimage captured by the camera 161 and the current image captured by thecamera 162, a parallax between the two images (hereinafter referred toas a stereo parallax) and supplies the stereo parallax to the decisionsection 182.

The decision section 182 decides whether or not it is necessary tocalibrate the cameras 161 and 162 on the basis of the movementparallaxes supplied from the movement parallax estimation sections 191and 192 and the stereo parallax supplied from the stereo parallaxestimation section 193. The decision section 182 supplies the decisionresult to the position detection section 45.

(Description of Processes Handled by Image Processing Apparatus)

Image processing handled by the image processing apparatus 180 depictedin FIG. 12 is similar to that depicted in FIG. 5 except that the cameras12 and 13 are replaced by the cameras 161 and 162 and except for thecalibration decision process. Therefore, only the calibration decisionprocess will be described below.

FIG. 13 is a flowchart describing the calibration decision process ofthe image processing apparatus 180.

In step S71 depicted in FIG. 13, the movement parallax estimationsection 191 of the estimation section 181 retains the image input fromthe camera 161, and the movement parallax estimation section 192 retainsthe image input from the camera 162.

In step S72, the movement parallax estimation section 191 estimates,using the retained past image of the camera 161 and the current image, amovement parallax of the current image and supplies the movementparallax to the decision section 182.

In step S73, the movement parallax estimation section 192 estimates,using the retained past image of the camera 162 and the current image, amovement parallax of the current image and supplies the movementparallax to the decision section 182.

In step S74, the stereo parallax estimation section 193 estimates astereo parallax using the current image of the camera 161 and thecurrent image of the camera 162 and supplies the stereo parallax to thedecision section 182.

In step S75, the decision section 182 decides whether the differencebetween at least one of the movement parallaxes supplied from themovement parallax estimation sections 191 and 192 and the stereoparallax supplied from the stereo parallax estimation section 193 isequal to a threshold or more.

If it is decided in step S75 that the difference is equal to thethreshold or more, the decision section 182 decides in step S76 that thecameras 161 and 162 need calibration and supplies the decision result tothe position detection section 45. Then, the calibration decisionprocess ends.

On the other hand, in a case where the difference between both of themovement parallaxes supplied from the movement parallax estimationsections 191 and 192 and the stereo parallax is smaller than thethreshold in step S75, the process proceeds to step S77. In step S77,the decision section 182 decides in step S77 that the cameras 161 and162 do not need calibration and supplies the decision result to theposition detection section 45.

Fifth Embodiment (Overview of Fifth Embodiment of Vehicle-Mounted CameraSystem)

FIG. 14 is a diagram illustrating an overview of a fifth embodiment ofthe vehicle-mounted camera system to which the present disclosure isapplied.

Of the components depicted in FIG. 14, the components identical to thosedepicted in FIG. 1 are denoted by the same reference numerals. Redundantdescription will be omitted as appropriate. It should be noted that FIG.14 is a view of the vehicle 11 as seen from above.

The configuration of a vehicle-mounted camera system 210 depicted inFIG. 14 differs from that of the vehicle-mounted camera system 10depicted in FIG. 1 in that four cameras 211 to 214 are mounted to thevehicle 11 in place of the cameras 12 and 13.

The cameras 211 to 214 are mounted to the front, right side, back, andleft side of the roof of the vehicle 11, respectively. The imageprocessing apparatus, not depicted, which is mounted to the vehicle 11performs the similar process as in the first to third embodiments forthe respective cameras 211 to 214 and calibrates the cameras 211 to 214.

It should be noted that four amounts of travel are found at this time.Therefore, whether or not it is necessary to calibrate the cameras 211to 214 may be decided on the basis of a distribution of four amounts oftravel without using the speed of the vehicle 11 measured by thespeedometer as in the third embodiment.

Sixth Embodiment

(Description of Computer to which Present Disclosure is Applied)

The series of processes of the image processing apparatus describedabove can be performed not only by hardware but also by software. In acase where the series of processes are performed by software, theprogram making up the software is installed to a computer. Here, thecomputer includes computers built into dedicated hardware andgeneral-purpose personal computers such as those capable of performingvarious functions as various programs are installed thereto.

FIG. 15 is a block diagram illustrating a hardware configuration exampleof a computer that performs the above series of processes usingprograms.

In a computer 300, a CPU (Central Processing Unit) 301, a ROM (Read OnlyMemory) 302, and a RAM (Random Access Memory) 303 are connected to eachother by a bus 304.

An I/O interface 305 is further connected to the bus 304. An inputsection 306, an output section 307, a storage section 308, acommunication section 309, and a drive 310 are connected to the I/Ointerface 305.

The input section 306 includes a keyboard, a mouse, a microphone, and soon. The output section 307 includes a display, a speaker, and so on. Thestorage section 308 includes a hard disk and a non-volatile memory. Thecommunication section 309 includes a network interface and so on. Thedrive 310 drives a removable medium 311 such as magnetic disk, opticaldisc, magneto-optical disk, or semiconductor memory.

In the computer 300 configured as described above, the series ofprocesses described above are performed as the CPU 301 loads, forexample, the program stored in the storage section 308 into the RAM 303via the I/O interface 305 and the bus 304 for execution.

The program executed by the computer 300 (CPU 301) can be provided, forexample, recorded on the removable medium 311 as a package media.Alternatively, the program can be provided via a wired or wirelesstransmission medium such as local area network, the Internet, or digitalsatellite broadcasting.

In the computer 300, the program can be installed to the storage section308 via the I/O interface 305 as the removable medium 311 is insertedinto the drive 310. Also, the program can be received by thecommunication section 309 via a wired or wireless transmission media andinstalled to the storage section 308. In addition to the above, theprogram can be installed, in advance, to the ROM 302 or the storagesection 308.

It should be noted that the program executed by the computer 300 may bea program that performs the processes chronologically according to thesequence described in the present specification, or in parallel, or at anecessary time as when the program is called.

Seventh Embodiment

(Description of Vehicle Control System to which Present Disclosure isApplied)

FIG. 16 is a block diagram illustrating an example of a schematicconfiguration of a vehicle control system to which the presentdisclosure is applied.

A vehicle control system 2000 includes a plurality of electronic controlunits connected via a communication network 2010. In the exampledepicted in FIG. 16, the vehicle control system 2000 includes adrive-system control unit 2100, a body-system control unit 2200, abattery control unit 2300, an out-vehicle information detection unit2400, an in-vehicle information detection unit 2500, and an integratedcontrol unit 2600. A communication network 2010 that connects theplurality of these control units may be a vehicle-mounted communicationnetwork such as CAN (Controller Area Network), LIN (Local InterconnectNetwork), LAN (Local Area Network) and FlexRay (registered trademark)compliant with an arbitrary standard.

Each control unit includes a microcomputer, a storage section, and adrive circuit. The microcomputer handles operations according to avariety of programs. The storage section stores programs executed by themicrocomputer or parameters used for various operations, and so on. Thedrive circuit drives various apparatuses to be controlled. Each controlunit includes not only a network I/F for communication with othercontrol units via the communication network 2010 but also acommunication I/F for communication with in- and out-vehicle apparatusesor sensors in a wired or wireless fashion. In FIG. 16, a microcomputer2610, a general-purpose communication I/F 2620, a dedicatedcommunication I/F 2630, a positioning section 2640, a beacon receptionsection 2650, an in-vehicle apparatus I/F 2660, an audio/image outputsection 2670, a vehicle-mounted network I/F 2680, and a storage section2690 are depicted as functional components of the integrated controlunit 2600. Other control units similarly include a microcomputer, acommunication I/F, a storage section, and so on.

The drive-system control unit 2100 controls the action of thedrive-system apparatuses of the vehicle in accordance with variousprograms. For example, the drive-system control unit 2100 functions as acontrol apparatus of a driving force generating apparatus for generatinga driving force of a vehicle such as internal combustion engine anddrive motor, a driving force transmission mechanism for transmitting adriving force to the wheels, a steering mechanism for adjusting thesteering angle of a vehicle, and a braking apparatus for generating abraking force of a vehicle. The drive-system control unit 2100 may alsohave functions as a control apparatus such as ABS (Antilock BrakeSystem) or an ESC (Electronic Stability Control).

A vehicle state detection section 2110 is connected to the drive-systemcontrol unit 2100. The vehicle state detection section 2110 includes,for example, at least one of a gyro sensor for detecting the angularspeed of axial rotational motion of a vehicle body, an accelerationsensor for detecting the acceleration of a vehicle, and a sensor fordetecting the amount of depression of the accelerator pedal, the amountof depression of the brake pedal, the steering angle of the steeringwheel, engine revolutions per minute, wheel rotational speed, and so on.The drive-system control unit 2100 performs operations using signalsinput from the vehicle state detection section 2110, thereby controllingthe internal combustion engine, the drive motor, the electric powersteering apparatus, or the brake apparatus.

The body-system control unit 2200 controls the action of variousapparatuses provided on the vehicle body in accordance with variousprograms. For example, the body-system control unit 2200 functions as acontrol apparatus of a keyless entry system, a smart key system, and apower window apparatus or various lamps such as headlights, rear lights,brake lamp, turn signals, or fog lamp. In this case, radio waves emittedfrom a portable transmitter that replaces a key or various switchsignals can be input to the body-system control unit 2200. Thebody-system control unit 2200 accepts these radio wave and signal inputsand controls the vehicle's door lock apparatus, power window apparatus,lamps, and so on.

The battery control unit 2300 controls a secondary battery 2310, a powersupply source of the drive motor, in accordance with various programs.For example, battery temperature, battery output voltage, remainingbattery charge, or other information is input to the battery controlunit 2300 from a battery apparatus having the secondary battery 2310.The battery control unit 2300 performs arithmetic processing using thesesignals, thereby controlling temperature adjustment of the secondarybattery 2310, a cooling apparatus provided on the battery apparatus, orother apparatus.

The out-vehicle information detection unit 2400 detects informationoutside a vehicle equipped with the vehicle control system 2000. Forexample, at least one of an imaging section 2410 and an out-vehicleinformation detection section 2420 is connected to the out-vehicleinformation detection unit 2400. The imaging section 2410 includes atleast one of a ToF (Time Of Flight) camera, a stereo camera, a monocularcamera, an infrared camera, and other cameras. The out-vehicleinformation detection section 2420 includes, for example, an environmentsensor that detects current weather or climate or a surroundinginformation detection sensor that detects other vehicles, obstacles,pedestrians, or others around the vehicle equipped with the vehiclecontrol system 2000.

The environment sensor may be, for example, one of a rain drop sensorthat detects rainy weather, a fog sensor that detects fog, a sunlightsensor that detects sunlight level, and a snow sensor that detectssnowfall. The surrounding information detection sensor may be one of anultrasonic sensor, a radar apparatus, and an LIDAR (Light Detection andRanging, Laser Imaging Detection and Ranging) apparatus. These imagingsection 2410 and out-vehicle information detection section 2420 may beincluded as separate sensors or apparatuses or as an integratedapparatus comprised of a plurality of sensors or apparatuses.

Here, FIG. 17 illustrates examples of installation positions of theimaging section 2410 and the out-vehicle information detection section2420. Imaging sections 2910, 2912, 2914, 2916, and 2918 are provided atleast one of a front nose, side mirrors, a rear bumper, a back door, anda top of a front glass in a compartment of a vehicle 2900. The imagingsection 2910 provided on the front nose and the imaging section 2918provided on the top of the front glass in the compartment acquire mainlyfront images of the vehicle 2900. The imaging sections 2912 and 2914provided on the side mirrors acquire mainly side images of the vehicle2900. The imaging section 2916 provided on the rear bumper or the backdoor acquires mainly a rear image of the vehicle 2900. The imagingsection 2918 provided on the top of the front glass in the compartmentis used mainly to detect vehicles ahead, pedestrians, obstacles, trafficlights, traffic signs, or driving lanes.

It should be noted that FIG. 17 illustrates examples of imaging rangesof the imaging sections 2910, 2912, 2914, and 2916. An imaging range ‘a’depicts the imaging range of the imaging section 2910 provided on thefront nose. Imaging ranges ‘b’ and ‘c’ depict the imaging ranges of theimaging sections 2912 and 2914 provided on the side mirrors. An imagingrange ‘d’ depicts the imaging range of the imaging section 2916 providedon the rear bumper or the back door. For example, superimposing imagedata, captured by the imaging sections 2910, 2912, 2914, and 2916, oneon top of the other, provides a bird's eye view image as seen from abovethe vehicle 2900.

Out-vehicle information detection sections 2920, 2922, 2924, 2926, 2928,and 2930 provided on the front, the rear, the sides, corners, and on thetop of the front glass in the compartment of the vehicle 2900 may be,for example, ultrasonic sensors or radar apparatuses. The out-vehicleinformation detection sections 2920, 2926, and 2930 provided on thefront nose, the rear bumper, the back door, and on the top of the frontglass in the compartment of the vehicle 2900 may be, for example, LIDARapparatuses. These out-vehicle information detection sections 2920 to2930 are used mainly to detect vehicles ahead, pedestrians, obstacles,or others.

A description will continue with reference back to FIG. 16. Theout-vehicle information detection unit 2400 causes the imaging section2410 to capture images outside the vehicle and receives captured imagedata. Also, the out-vehicle information detection unit 2400 receivesdetection information from the connected out-vehicle informationdetection section 2420. In a case where the out-vehicle informationdetection section 2420 is an ultrasonic sensor, a radar apparatus, or anLIDAR apparatus, the out-vehicle information detection unit 2400 causesan ultrasonic wave, an electromagnetic wave, or other wave to be emittedand receives information about a received reflected wave. Theout-vehicle information detection unit 2400 may perform an objectdetection process for detecting persons, vehicles, obstacles, signs,characters on the road, or others or a distance detection process on thebasis of the received information. The out-vehicle information detectionunit 2400 may perform an environment recognition process for detectingrainfall, fog, road surface condition or others on the basis of thereceived information. The out-vehicle information detection unit 2400may calculate a distance to an object outside the vehicle on the basisof the received information.

Also, the out-vehicle information detection unit 2400 may perform animage recognition process for recognizing persons, vehicles, obstacles,signs, characters on the road, or others or a distance detection processon the basis of the received information. The out-vehicle informationdetection unit 2400 may generate a bird's eye view image or a panoramicimage by performing distortion correction, position alignment, or otherprocess on the received image data and combining the data with imagedata captured by the different imaging section 2410. The out-vehicleinformation detection unit 2400 may perform a viewpoint conversionprocess using image data captured by the different imaging section 2410.

The in-vehicle information detection unit 2500 detects in-vehicleinformation. For example, a driver state detection section 2510 thatdetects the driver's state is connected to the in-vehicle informationdetection unit 2500. The driver state detection section 2510 may be acamera that images the driver, a biological sensor that detectsbiological information of the driver, a microphone that collects audioin the compartment, or other apparatus. A biological sensor is provided,for example, on a seat surface, the steering wheel, or other location todetect biological information of a passenger sitting on the seat or thedriver holding the steering wheel. The in-vehicle information detectionunit 2500 may calculate fatigue level or concentration level of thedriver on the basis of detection information input from the driver statedetection section 2510. Whether the driver is drowsing may be decided.The in-vehicle information detection unit 2500 may subject a collectedaudio signal to a noise canceling process or other process.

The integrated control unit 2600 controls the actions within the vehiclecontrol system 2000 as a whole in accordance with various programs. Aninput section 2800 is connected to the integrated control unit 2600. Theinput section 2800 is realized, for example, by a touch panel, buttons,a microphone, switches, levers, or others on which input operation canbe made. The input section 2800 may be, for example, a remote controlapparatus on the basis of infrared radiation or other radio waves or anexternal connection apparatus such as mobile phone, PDA (PersonalDigital Assistant), or others capable of manipulating the vehiclecontrol system 2000. The input section 2800 may be, for example, acamera, and in this case, a passenger can input information by gesture.Further, the input section 2800 may include an input control circuitthat generates an input signal on the basis of the above informationinput by a passenger or others by using the input section 2800 andoutputs the input signal to the integrated control unit 2600. Passengersand so on operate the input section 2800 to input various data to thevehicle control system 2000 and instruct the vehicle control system 2000to process data.

The storage section 2690 may include a RAM (Random Access Memory) thatstores various programs executed by a microcomputer and a ROM (Read OnlyMemory) that stores various parameters, operation results, sensorvalues, and other data. Also, the storage section 2690 may be realizedby a magnetic storage device such as HDD (Hard Disc Drive),semiconductor storage device, optical storage device, magneto-opticalstorage device, or other device.

The general-purpose communication I/F 2620 is a general-purposecommunication interface that intermediates communication with variousapparatuses existing in an outside environment 2750. A cellularcommunication protocol such as GSM (registered trademark) (Global Systemof Mobile communications), WiMAX, LTE (Long Term Evolution), or LTE-A(LTE-Advanced) or other wireless communication protocol such as wirelessLAN (also referred to as Wi-Fi (registered trademark)) may beimplemented in the general-purpose communication I/F 2620. Thegeneral-purpose communication I/F 2620 may connect, for example, to anapparatus (e.g., application server or control server) existing on anexternal network (e.g., Internet, cloud network, or carrier's ownnetwork) via a base station and an access point. Also, thegeneral-purpose communication I/F 2620 may connect to a terminalexisting near the vehicle (e.g., pedestrian's or shop's terminal or MTC(Machine Type Communication) terminal) by using, for example, P2P (PeerTo Peer) technology.

The dedicated communication I/F 2630 is a communication protocol thatsupports a communication protocol developed to be used in vehicles. Astandard protocol such as WAVE (Wireless Access in Vehicle Environment),a combination of IEEE802.11p, a lower layer, and IEEE1609, an upperlayer, or DSRC (Dedicated Short Range Communications), for example, maybe implemented in the dedicated communication I/F 2630. The dedicatedcommunication I/F 2630 typically carries out V2X communication, aconcept that includes one or more of vehicle to vehicle communication,vehicle to infrastructure communication, and vehicle to pedestriancommunication.

The positioning section 2640 carries out positioning by receiving a GNSSsignal (e.g., GPS signal from GPS (Global Positioning System) satellite)from a GNSS (Global Navigation Satellite System) satellite and generatesposition information including longitude, latitude, and altitude of thevehicle. It should be noted that the positioning section 2640 mayidentify the current position by exchanging signals with wireless accesspoints or acquire position information from a terminal such as mobilephone, PHS, or smartphone.

The beacon reception section 2650 acquires current position, trafficjams, road closures, required time, or other information by receivingradio waves or electromagnetic waves emitted from wireless stations orother apparatuses installed on roads. It should be noted that thefunctions of the beacon reception section 2650 may be included in thededicated communication I/F 2630.

The in-vehicle apparatus I/F 2660 is a communication interface thatintermediates communication between the microcomputer 2610 and variouspieces of equipment existing in the vehicle. The in-vehicle apparatusI/F 2660 may establish wireless connection by using a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), NFC (Near Field Communication), or WUSB (Wireless USB).Also, the in-vehicle apparatus I/F 2660 may establish wired connectionby using a connection terminal which is not depicted (and a cable ifrequired). The in-vehicle apparatus I/F 2660 exchanges control signalsor data signals, for example, with a mobile apparatus or a wearableapparatus of a passenger, or an information apparatus carried into orinstalled in the vehicle.

The vehicle-mounted network I/F 2680 is an interface that intermediatescommunication between the microcomputer 2610 and the communicationnetwork 2010. The vehicle-mounted network I/F 2680 sends and receivessignals and others according to a given protocol supported by thecommunication network 2010.

The microcomputer 2610 of the integrated control unit 2600 controls thevehicle control system 2000 in accordance with various programs based oninformation acquired via at least one of the general-purposecommunication I/F 2620, the dedicated communication I/F 2630, thepositioning section 2640, the beacon reception section 2650, thein-vehicle apparatus I/F 2660, and the vehicle-mounted network I/F 2680.For example, the microcomputer 2610 may calculate a control target valueof the driving force generating apparatus, the steering mechanism, orthe brake apparatus on the basis of in-vehicle and out-vehicleinformation acquired and output a control command to the drive-systemcontrol unit 2100. For example, the microcomputer 2610 may performcooperative control for vehicle collision avoidance, or impactalleviation, follow-up traveling on the basis of vehicle-to-vehicledistance, constant vehicle speed traveling, autonomous driving, and soon.

The microcomputer 2610 may create local map information includinginformation around the current position of the vehicle on the basis ofinformation acquired via at least one of the general-purposecommunication I/F 2620, the dedicated communication I/F 2630, thepositioning section 2640, the beacon reception section 2650, thein-vehicle apparatus I/F 2660, and the vehicle-mounted network I/F 2680.Also, the microcomputer 2610 may predict risks such as collision of thevehicle, approaching pedestrian, and entry into a closed road andgenerate a warning signal. A warning signal may be a signal that causesa warning tone to be produced or a warning lamp to be lit.

The audio/image output section 2670 sends at least either an audio orimage output signal to an output apparatus that is capable of visuallyor auditorily notifying information to the vehicle's passenger oroutside of the vehicle. In the example depicted in FIG. 16, an audiospeaker 2710, a display section 2720, and an instrument panel 2730 aredepicted as output apparatuses. The display section 2720 may include,for example, at least one of an on-board display and a head-up display.The display section 2720 may include an AR (Augmented Reality) displayfunction. The output apparatus may be an apparatus other than the abovesuch as headphone, projector, or lamp. In a case where the outputapparatus is a display apparatus, the display apparatus visuallydisplays results obtained by various processes performed by themicrocomputer 2610 or information received from other control units invarious forms such as text, image, table, and graph. Also, in a casewhere the output apparatus is an audio output apparatus, the audiooutput apparatus converts an audio signal made up of audio data,acoustic data, or other data into an analog signal and auditorilyoutputs the analog signal.

It should be noted that, in the example depicted in FIG. 16, at leasttwo control units connected via the communication network 2010 may becombined into a single control unit. Alternatively, each control unitmay include a plurality of control units. Further, the vehicle controlsystem 2000 may include a separate control unit that is not depicted.Also, in the description given above, some or all of the functionsassumed by any of the control units may be assumed by other controlunit. That is, as long as information is sent and received via thecommunication network 2010, given arithmetic processing may be performedby one of the control units. Similarly, a sensor or apparatus connectedto one of the control units may be connected to other control unit sothat the plurality of control units mutually send and receive detectioninformation via the communication network 2010.

In the motor vehicle control system 2000 configured as described above,the functions of the image processing apparatus of the presentapplication are provided in the integrated control unit 2600. It shouldbe noted that at least some of the functions of the image processingapparatus of the present application may be realized by a module for theintegrated control unit 2600 (e.g., integrated circuit module configuredon a single die). Also, the image processing apparatus of the presentapplication may be realized by a plurality of control units.

In the present specification, a system refers to a set of a plurality ofcomponents (e.g., apparatuses, modules (parts)), and it does not matterwhether or not all the components are accommodated in the same housing.Therefore, a plurality of apparatuses accommodated in different housingsand connected via a network and a single apparatus having a plurality ofmodules accommodated in a single housing are both systems.

It should be noted that the effect described in the presentspecification is merely illustrative and is not limited and that theremay be additional effects.

Also, embodiments of the present disclosure are not limited to thosedescribed above and can be modified in various ways without departingfrom the gist of the present disclosure.

For example, the number and arrangement of cameras (imaging sections)making up the vehicle-mounted camera system (vehicle control system) arenot limited to the number and arrangement described above. Also, thepresent technology is also applicable to vehicle-mounted camera systemsmounted not only to motor vehicles but also vehicles including electricvehicles, hybrid electric vehicles, and motorcycles.

Also, a method of estimating a 3D position and orientation of a knownobject is not limited to that described above. Further, cameras may becalibrated on the basis only on a 3D position of a known object.

It should be noted that the present disclosure can have the followingconfigurations:

(1)

An image processing apparatus including:

a decision section configured to decide, using a first image captured bya first imaging section at a first time and a second image captured bythe first imaging section at a second time later than the first time,whether or not it is necessary to calibrate the first imaging section.

(2)

The image processing apparatus of feature (1), further including:

an amount-of-travel estimation section configured to estimate an amountof travel of a three dimensional position of the first imaging sectionusing the first and second images, in which

the decision section decides whether or not the calibration is necessaryon the basis of the amount of travel estimated by the amount-of-travelestimation section.

(3)

The image processing apparatus of feature (2), in which

the amount-of-travel estimation section estimates amounts of travel ofthe three dimensional position and orientation of the first imagingsection.

(4)

The image processing apparatus of feature (3), in which

the amount-of-travel estimation section estimates, on the basis of athree dimensional position of a feature point of the first image and aposition of the feature point on the second image, amounts of travel ofthe three dimensional position and orientation of the feature point.

(5)

The image processing apparatus of feature (4), further including:

a parallax detection section configured to detect a stereo parallax ofthe feature point of the first image using a third image captured by asecond imaging section at the first time; and

a position calculation section configured to calculate a threedimensional position of the feature point of the first image on thebasis of the stereo parallax detected by the parallax detection section.

(6)

The image processing apparatus of feature (4), further including:

a parallax detection section configured to detect a movement parallax ofthe feature point of the first image using the first image and a thirdimage captured by the first imaging section at a third time differentfrom the first time; and

a position calculation section configured to calculate a threedimensional position of the feature point of the first image on thebasis of the movement parallax detected by the parallax detectionsection.

(7)

The image processing apparatus of any one of features (2) to (6), inwhich

the first imaging section is mounted to a vehicle, and the decisionsection decides whether or not the calibration is necessary on the basisof the amount of travel and a speed of the vehicle measured by aspeedometer of the vehicle.

(8)

The image processing apparatus of feature (1), further including:

a first movement parallax estimation section configured to estimate amovement parallax of the first image using the first and second images;

a second movement parallax estimation section configured to estimate amovement parallax of the second image using a third image captured by asecond imaging section at the first time and a fourth image captured bythe second imaging section at the second time; and

a stereo parallax estimation section configured to estimate a stereoparallax between the first and third images using the first and thirdimages, in which

the decision section decides whether or not it is necessary to calibratethe first and second imaging sections on the basis of the movementparallax of the first image estimated by the first movement parallaxestimation section, the movement parallax of the second image estimatedby the second movement parallax estimation section, and the stereoparallax estimated by the stereo parallax estimation section.

(9)

The image processing apparatus of feature (8), in which

the first and second imaging sections are mounted to a vehicle.

(10)

The image processing apparatus of feature (9), in which

directions of optical axes of the first and second imaging sections areapproximately vertical to a direction of travel of the vehicle.

(11)

An image processing method, by an image processing apparatus, including:

a decision step of deciding, using a first image captured by a firstimaging section at a first time and a second image captured by the firstimaging section at a second time later than the first time, whether ornot it is necessary to calibrate the first imaging section.

(12)

A program causing a computer to function as:

a decision section configured to decide, using a first image captured bya first imaging section at a first time and a second image captured bythe first imaging section at a second time later than the first time,whether or not it is necessary to calibrate the first imaging section.

REFERENCE SIGNS LIST

11 Vehicle, 12, 13 Camera, 21 Road sign, 40 Image processing apparatus,41 Amount-of-travel estimation section, 42 Decision section, 43Amount-of-travel estimation section, 44 Decision section, 46 Correctionsection, 64 Parallax detection section, 65 Position calculation section,83, 86 Estimation section, 87 Recognition section, 111 Number plate,123, 126 Estimation section, 127 Recognition section, 141 Camera, 161,162 Camera, 180 Image processing apparatus, 182 Decision section, 191,192 Movement parallax estimation section, 193 Stereo parallax estimationsection

1. An image processing apparatus comprising: a decision sectionconfigured to decide, using a first image captured by a first imagingsection at a first time and a second image captured by the first imagingsection at a second time later than the first time, whether or not it isnecessary to calibrate the first imaging section.
 2. The imageprocessing apparatus of claim 1, further comprising: an amount-of-travelestimation section configured to estimate an amount of travel of a threedimensional position of the first imaging section using the first andsecond images, wherein the decision section decides whether or not thecalibration is necessary on the basis of the amount of travel estimatedby the amount-of-travel estimation section.
 3. The image processingapparatus of claim 2, wherein the amount-of-travel estimation sectionestimates amounts of travel of the three dimensional position andorientation of the first imaging section.
 4. The image processingapparatus of claim 3, wherein the amount-of-travel estimation sectionestimates, on the basis of a three dimensional position of a featurepoint of the first image and a position of the feature point on thesecond image, amounts of travel of the three dimensional position andorientation of the feature point.
 5. The image processing apparatus ofclaim 4, further comprising: a parallax detection section configured todetect a stereo parallax of the feature point of the first image using athird image captured by a second imaging section at the first time; anda position calculation section configured to calculate a threedimensional position of the feature point of the first image on thebasis of the stereo parallax detected by the parallax detection section.6. The image processing apparatus of claim 4, further comprising: aparallax detection section configured to detect a movement parallax ofthe feature point of the first image using the first image and a thirdimage captured by the first imaging section at a third time differentfrom the first time; and a position calculation section configured tocalculate a three dimensional position of the feature point of the firstimage on the basis of the movement parallax detected by the parallaxdetection section.
 7. The image processing apparatus of claim 2, whereinthe first imaging section is mounted to a vehicle, and the decisionsection decides whether or not the calibration is necessary on the basisof the amount of travel and a speed of the vehicle measured by aspeedometer of the vehicle.
 8. The image processing apparatus of claim1, further comprising: a first movement parallax estimation sectionconfigured to estimate a movement parallax of the first image using thefirst and second images; a second movement parallax estimation sectionconfigured to estimate a movement parallax of the second image using athird image captured by a second imaging section at the first time and afourth image captured by the second imaging section at the second time;and a stereo parallax estimation section configured to estimate a stereoparallax between the first and third images using the first and thirdimages, wherein the decision section decides whether or not it isnecessary to calibrate the first and second imaging sections on thebasis of the movement parallax of the first image estimated by the firstmovement parallax estimation section, the movement parallax of thesecond image estimated by the second movement parallax estimationsection, and the stereo parallax estimated by the stereo parallaxestimation section.
 9. The image processing apparatus of claim 8,wherein the first and second imaging sections are mounted to a vehicle.10. The image processing apparatus of claim 9, wherein directions ofoptical axes of the first and second imaging sections are approximatelyvertical to a direction of travel of the vehicle.
 11. An imageprocessing method, by an image processing apparatus, comprising: adecision step of deciding, using a first image captured by a firstimaging section at a first time and a second image captured by the firstimaging section at a second time later than the first time, whether ornot it is necessary to calibrate the first imaging section.
 12. Aprogram causing a computer to function as: a decision section configuredto decide, using a first image captured by a first imaging section at afirst time and a second image captured by the first imaging section at asecond time later than the first time, whether or not it is necessary tocalibrate the first imaging section.